logical: add initial/catchup scan metrics to offline scan #155274
Conversation
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=95b789b1d594ac6b74f377aa7555d8d98d63240d&v=4){kind=avatar}
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
This change is |
Screenshots
I think some additional work could probably be done for the job messages to make it more clear that we've moved from offline scan to the steady state. Just a one-liner adding a job message when the offline processor is complete is probably sufficient. |
There was a problem hiding this comment.
Pull Request Overview
This PR enhances the LDR offline scan processor to emit checkpoint range statistics and update the logical_replication.scanning_ranges and logical_replication.catchup_ranges metrics during fast initial scans.
Key changes:
- Added range statistics collection to the offline initial scan processor
- Refactored error handling in the Next() method for better maintainability
- Enhanced checkpoint processing to include range statistics
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pkg/crosscluster/logical/offline_initial_scan_processor.go | Added range statistics channel, refactored error handling, and enhanced checkpoint processing to emit range stats |
| pkg/crosscluster/logical/logical_replication_job.go | Added tracking of write processor count for metrics |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
This patch teaches the LDR offline scan processor to emit checkpoint range stats and update the `logical_replication.scanning_ranges` and `logical_replication.catchup_ranges` metrics. Informs: cockroachdb#152273 Release note: LDR now updates the `logical_replication.scanning_ranges` and `logical_replication.catchup_ranges` metrics during fast initial scan.
kev-cao
force-pushed
the
logical/offline-metrics
branch
from
7c54aaa to
02bdcbc
Compare
| sourceSpans []roachpb.Span | ||
| partitionPgUrls []string | ||
| destTableBySrcID map[descpb.ID]dstTableMetadata | ||
| sourceSpans []roachpb.Span |
There was a problem hiding this comment.
what autoformatter do you use? is it more than crlfmt?
There was a problem hiding this comment.
It should just be crlfmt 🤔
There was a problem hiding this comment.
Oh this particular formatting case I think happened because of the comment — I believe crlfmt only indents types for fields that are adjacent, and a comment breaks that adjacency.
|
TFTR! bors r=msbutler |
|
Build succeeded: |
3 participants
This patch teaches the LDR offline scan processor to emit checkpoint range stats and update the
logical_replication.scanning_rangesandlogical_replication.catchup_rangesmetrics.Informs: #152273
Release note: LDR now updates the
logical_replication.scanning_rangesandlogical_replication.catchup_rangesmetrics during fast initial scan.